System and method for providing more appropriate question/answer responses based upon profiles

ABSTRACT

A method, computer program product, and computer system for receiving, by a computing device, a question from a user. A first answer provided by a first subject matter expert is identified. A second answer provided by a second subject matter expert is identified. It is determined that a profile of the user matches a first profile of the first subject matter expert more than a second profile of the second subject matter expert. The first answer provided by the first subject matter expert is sent to the user with a preference over the second answer provided by the second subject matter expert based upon, at least in part, determining that the profile of the user matches the first profile of the first subject matter expert more than the second profile of the second subject matter expert.

RELATED APPLICATIONS

The subject application is a continuation application of U.S. patentapplication Ser. No. 14/641,954, filed on Mar. 9, 2015, the entirecontents of which are herein incorporated by reference.

BACKGROUND

Some traditional QA system may engage Subject Matter Experts (SMEs) tocreate the ground truth (candidate QA pairs) as part of QA systemtraining. Generally, the process for this may have a unique set ofquestions that are given to each SME to create this set. In othersystems, a collaborative model may be used where each SME is given thesame questions and then through a group decision (e.g., voting) come upwith the most popular answers that are used to create the set. However,there may be example situations where, e.g., there are two or moregenerally equally good answers that may vary depending on which end useris asking the question.

BRIEF SUMMARY OF DISCLOSURE

In one example implementation, a method, performed by one or morecomputing devices, may include but is not limited to receiving, by acomputing device, a question from a user. A first answer provided by afirst subject matter expert may be identified. A second answer providedby a second subject matter expert may be identified. It may bedetermined that a profile of the user matches a first profile of thefirst subject matter expert more than a second profile of the secondsubject matter expert. The first answer provided by the first subjectmatter expert may be sent to the user with a preference over the secondanswer provided by the second subject matter expert based upon, at leastin part, determining that the profile of the user matches the firstprofile of the first subject matter expert more than the second profileof the second subject matter expert.

One or more of the following example features may be included.Determining that the profile of the user matches the first profile ofthe first subject matter expert more than the second profile of thesecond subject matter expert may include applying a weight to one ormore features of the profile. Determining that the profile of the usermatches the first profile of the first subject matter expert more thanthe second profile of the second subject matter expert may includecomparing the profile of the user with the first profile of the firstsubject matter expert, and may include comparing the profile of the userwith the second profile of the second subject matter expert. At leastone of the user profile, the first profile, and the second profile mayinclude a personality model. At least one of the user profile, the firstprofile, and the second profile may include an experience model. Atleast one of the user profile, the first profile, and the second profilemay include a trait model. The preference may include a ranking of thefirst answer provided by the first subject matter expert higher than thesecond answer provided by the second subject matter expert.

In another example implementation, a computing system includes aprocessor and a memory configured to perform operations that may includebut are not limited to receiving a question from a user. A first answerprovided by a first subject matter expert may be identified. A secondanswer provided by a second subject matter expert may be identified. Itmay be determined that a profile of the user matches a first profile ofthe first subject matter expert more than a second profile of the secondsubject matter expert. The first answer provided by the first subjectmatter expert may be sent to the user with a preference over the secondanswer provided by the second subject matter expert based upon, at leastin part, determining that the profile of the user matches the firstprofile of the first subject matter expert more than the second profileof the second subject matter expert.

One or more of the following example features may be included.Determining that the profile of the user matches the first profile ofthe first subject matter expert more than the second profile of thesecond subject matter expert may include applying a weight to one ormore features of the profile. Determining that the profile of the usermatches the first profile of the first subject matter expert more thanthe second profile of the second subject matter expert may includecomparing the profile of the user with the first profile of the firstsubject matter expert, and may include comparing the profile of the userwith the second profile of the second subject matter expert. At leastone of the user profile, the first profile, and the second profile mayinclude a personality model. At least one of the user profile, the firstprofile, and the second profile may include an experience model. Atleast one of the user profile, the first profile, and the second profilemay include a trait model. The preference may include a ranking of thefirst answer provided by the first subject matter expert higher than thesecond answer provided by the second subject matter expert.

In another example implementation, a computer program product resides ona computer readable storage medium that has a plurality of instructionsstored on it. When executed by a processor, the instructions cause theprocessor to perform operations that may include but are not limited toreceiving a question from a user. A first answer provided by a firstsubject matter expert may be identified. A second answer provided by asecond subject matter expert may be identified. It may be determinedthat a profile of the user matches a first profile of the first subjectmatter expert more than a second profile of the second subject matterexpert. The first answer provided by the first subject matter expert maybe sent to the user with a preference over the second answer provided bythe second subject matter expert based upon, at least in part,determining that the profile of the user matches the first profile ofthe first subject matter expert more than the second profile of thesecond subject matter expert.

One or more of the following example features may be included.Determining that the profile of the user matches the first profile ofthe first subject matter expert more than the second profile of thesecond subject matter expert may include applying a weight to one ormore features of the profile. Determining that the profile of the usermatches the first profile of the first subject matter expert more thanthe second profile of the second subject matter expert may includecomparing the profile of the user with the first profile of the firstsubject matter expert, and may include comparing the profile of the userwith the second profile of the second subject matter expert. At leastone of the user profile, the first profile, and the second profile mayinclude a personality model. At least one of the user profile, the firstprofile, and the second profile may include an experience model. Atleast one of the user profile, the first profile, and the second profilemay include a trait model. The preference may include a ranking of thefirst answer provided by the first subject matter expert higher than thesecond answer provided by the second subject matter expert.

The details of one or more example implementations are set forth in theaccompanying drawings and the description below. Other features andadvantages will become apparent from the description, the drawings, andthe claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example diagrammatic view of a response process coupled toa distributed computing network according to one or more exampleimplementations of the disclosure;

FIG. 2 is an example diagrammatic view of a client electronic device ofFIG. 1 according to one or more example implementations of thedisclosure;

FIG. 3 is an example flowchart of the response process of FIG. 1according to one or more example implementations of the disclosure;

FIG. 4 is an example diagrammatic view of a screen image displayed bythe response process of FIG. 1 according to one or more exampleimplementations of the disclosure;

FIG. 5 is an example diagrammatic view of a screen image displayed bythe response process of FIG. 1 according to one or more exampleimplementations of the disclosure; and

FIG. 6 is an example diagrammatic view of the response process of FIG. 1according to one or more example implementations of the disclosure.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

System Overview:

As will be appreciated by one skilled in the art, aspects of the presentdisclosure may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present disclosure may take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present disclosure are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Referring now to FIG. 1, there is shown response process 10 that mayreside on and may be executed by a computer (e.g., computer 12), whichmay be connected to a network (e.g., network 14) (e.g., the internet ora local area network). Examples of computer 12 (and/or one or more ofthe client electronic devices noted below) may include, but are notlimited to, a personal computer(s), a laptop computer(s), mobilecomputing device(s), a server computer, a series of server computers, amainframe computer(s), or a computing cloud(s). Computer 12 may executean operating system, for example, but not limited to, Microsoft®Windows®; Mac® OS X®; Red Hat® Linux®, or a custom operating system.(Microsoft and Windows are registered trademarks of MicrosoftCorporation in the United States, other countries or both; Mac and OS Xare registered trademarks of Apple Inc. in the United States, othercountries or both; Red Hat is a registered trademark of Red HatCorporation in the United States, other countries or both; and Linux isa registered trademark of Linus Torvalds in the United States, othercountries or both).

As will be discussed below in greater detail, response process 10 mayreceive, by a computing device, a question from a user. A first answerprovided by a first subject matter expert may be identified. A secondanswer provided by a second subject matter expert may be identified. Itmay be determined that a profile of the user matches a first profile ofthe first subject matter expert more than a second profile of the secondsubject matter expert. The first answer provided by the first subjectmatter expert may be sent to the user with a preference over the secondanswer provided by the second subject matter expert based upon, at leastin part, determining that the profile of the user matches the firstprofile of the first subject matter expert more than the second profileof the second subject matter expert.

The instruction sets and subroutines of response process 10, which maybe stored on storage device 16 coupled to computer 12, may be executedby one or more processors (not shown) and one or more memoryarchitectures (not shown) included within computer 12. Storage device 16may include but is not limited to: a hard disk drive; a flash drive, atape drive; an optical drive; a RAID array; a random access memory(RAM); and a read-only memory (ROM).

Network 14 may be connected to one or more secondary networks (e.g.,network 18), examples of which may include but are not limited to: alocal area network; a wide area network; or an intranet, for example.

Computer 12 may include a data store, such as a database (e.g.,relational database, object-oriented database, triplestore database,etc.) and may be located within any suitable memory location, such asstorage device 16 coupled to computer 12. Any data described throughoutthe present disclosure may be stored in the data store. In someimplementations, computer 12 may utilize a database management systemsuch as, but not limited to, “My Structured Query Language” (MySQL®) inorder to provide multi-user access to one or more databases, such as theabove noted relational database. The data store may also be a customdatabase, such as, for example, a flat file database or an XML database.Any other form(s) of a data storage structure and/or organization mayalso be used. Response process 10 may be a component of the data store,a stand alone application that interfaces with the above noted datastore and/or an applet/application that is accessed via clientapplications 22, 24, 26, 28. The above noted data store may be, in wholeor in part, distributed in a cloud computing topology. In this way,computer 12 and storage device 16 may refer to multiple devices, whichmay also be distributed throughout the network.

Computer 12 may execute a Question Answering (QA) system application,such as (e.g., QA application 20), examples of which may include, butare not limited to, e.g., the IBM Watson™ application or other QAapplication, a search engine application, a natural language processing(NLP) application, or other application that allows for the answering ofquestions posed by a user by querying stored information, e.g., in adata store. Response process 10 and/or QA application 20 may be accessedvia client applications 22, 24, 26, 28. Response process 10 may be astand alone application, or may be anapplet/application/script/extension that may interact with and/or beexecuted within QA application 20, a component of QA application 20,and/or one or more of client applications 22, 24, 26, 28. QA application20 may be a stand alone application, or may be anapplet/application/script/extension that may interact with and/or beexecuted within response process 10, a component of response process 10,and/or one or more of client applications 22, 24, 26, 28. One or more ofclient applications 22, 24, 26, 28 may be a stand alone application, ormay be an applet/application/script/extension that may interact withand/or be executed within and/or be a component of response process 10and/or QA application 20. Examples of client applications 22, 24, 26, 28may include, but are not limited to, e.g., the IBM Watson™ applicationor other QA application, a search engine application, a natural languageprocessing (NLP) application, or other application that allows for theanswering of questions posed by a user by querying stored information,e.g., in a data store, a standard and/or mobile web browser, an emailclient application, a textual and/or a graphical user interface, acustomized web browser, a plugin, an Application Programming Interface(API), or a custom application. The instruction sets and subroutines ofclient applications 22, 24, 26, 28, which may be stored on storagedevices 30, 32, 34, 36, coupled to client electronic devices 38, 40, 42,44, may be executed by one or more processors (not shown) and one ormore memory architectures (not shown) incorporated into clientelectronic devices 38, 40, 42, 44.

Storage devices 30, 32, 34, 36, may include but are not limited to: harddisk drives; flash drives, tape drives; optical drives; RAID arrays;random access memories (RAM); and read-only memories (ROM). Examples ofclient electronic devices 38, 40, 42, 44 (and/or computer 12) mayinclude, but are not limited to, a personal computer (e.g., clientelectronic device 38), a laptop computer (e.g., client electronic device40), a smart/data-enabled, cellular phone (e.g., client electronicdevice 42), a notebook computer (e.g., client electronic device 44), atablet (not shown), a server (not shown), a television (not shown), asmart television (not shown), a media (e.g., video, photo, etc.)capturing device (not shown), and a dedicated network device (notshown). Client electronic devices 38, 40, 42, 44 may each execute anoperating system, examples of which may include but are not limited to,Android™, Apple® iOS®, Mac® OS X®; Red Hat® Linux®, or a customoperating system.

One or more of client applications 22, 24, 26, 28 may be configured toeffectuate some or all of the functionality of response process 10 (andvice versa). Accordingly, response process 10 may be a purelyserver-side application, a purely client-side application, or a hybridserver-side/client-side application that is cooperatively executed byone or more of client applications 22, 24, 26, 28 and/or responseprocess 10.

One or more of client applications 22, 24, 26, 28 may be configured toeffectuate some or all of the functionality of QA application 20 (andvice versa). Accordingly, QA application 20 may be a purely server-sideapplication, a purely client-side application, or a hybridserver-side/client-side application that is cooperatively executed byone or more of client applications 22, 24, 26, 28 and/or QA application20. As one or more of client applications 22, 24, 26, 28, responseprocess 10, and QA application 20, taken singly or in any combination,may effectuate some or all of the same functionality, any description ofeffectuating such functionality via one or more of client applications22, 24, 26, 28, response process 10, QA application 20, or combinationthereof, and any described interaction(s) between one or more of clientapplications 22, 24, 26, 28, response process 10, QA application 20, orcombination thereof to effectuate such functionality, should be taken asan example only and not to limit the scope of the disclosure.

Users 46, 48, 50, 52 may access computer 12 and response process 10(e.g., using one or more of client electronic devices 38, 40, 42, 44)directly through network 14 or through secondary network 18. Further,computer 12 may be connected to network 14 through secondary network 18,as illustrated with phantom link line 54. Response process 10 mayinclude one or more user interfaces, such as browsers and textual orgraphical user interfaces, through which users 46, 48, 50, 52 may accessresponse process 10.

The various client electronic devices may be directly or indirectlycoupled to network 14 (or network 18). For example, client electronicdevice 38 is shown directly coupled to network 14 via a hardwirednetwork connection. Further, client electronic device 44 is showndirectly coupled to network 18 via a hardwired network connection.Client electronic device 40 is shown wirelessly coupled to network 14via wireless communication channel 56 established between clientelectronic device 40 and wireless access point (i.e., WAP) 58, which isshown directly coupled to network 14. WAP 58 may be, for example, anIEEE 802.11a, 802.11b, 802.11g, Wi-Fi®, and/or Bluetooth™ device that iscapable of establishing wireless communication channel 56 between clientelectronic device 40 and WAP 58. Client electronic device 42 is shownwirelessly coupled to network 14 via wireless communication channel 60established between client electronic device 42 and cellularnetwork/bridge 62, which is shown directly coupled to network 14.

Some or all of the IEEE 802.11x specifications may use Ethernet protocoland carrier sense multiple access with collision avoidance (i.e.,CSMA/CA) for path sharing. The various 802.11x specifications may usephase-shift keying (i.e., PSK) modulation or complementary code keying(i.e., CCK) modulation, for example. Bluetooth™ is a telecommunicationsindustry specification that allows, e.g., mobile phones, computers,smart phones, and other electronic devices to be interconnected using ashort-range wireless connection. Other forms of interconnection (e.g.,Near Field Communication (NFC)) may also be used.

Referring also to FIG. 2, there is shown a diagrammatic view of clientelectronic device 38. While client electronic device 38 is shown in thisfigure, this is for illustrative purposes only and is not intended to bea limitation of this disclosure, as other configurations are possible.For example, any computing device capable of executing, in whole or inpart, response process 10 may be substituted for client electronicdevice 38 within FIG. 2, examples of which may include but are notlimited to computer 12 and/or client electronic devices 40, 42, 44.

Client electronic device 38 may include a processor and/ormicroprocessor (e.g., microprocessor 200) configured to, e.g., processdata and execute the above-noted code/instruction sets and subroutines.Microprocessor 200 may be coupled via a storage adaptor (not shown) tothe above-noted storage device(s) (e.g., storage device 30). An I/Ocontroller (e.g., I/O controller 202) may be configured to couplemicroprocessor 200 with various devices, such as keyboard 206,pointing/selecting device (e.g., mouse 208), custom device (e.g., device215), USB ports (not shown), and printer ports (not shown). A displayadaptor (e.g., display adaptor 210) may be configured to couple display212 (e.g., CRT or LCD monitor(s)) with microprocessor 200, while networkcontroller/adaptor 214 (e.g., an Ethernet adaptor) may be configured tocouple microprocessor 200 to the above-noted network 14 (e.g., theInternet or a local area network).

Question Answering (QA) systems, such as the IBM Watson™ system, mayinclude an application of advanced natural language processing,information retrieval, knowledge representation and reasoning, andmachine learning technologies to the field of open domain questionanswering. The IBM Watson™ system may be built on IBM's DeepQAtechnology (or other compatible technology) used for hypothesisgeneration, massive evidence gathering, analysis, and scoring. The IBMWatson™ system may, for instance, take an input question, analyze it,decompose the question into constituent parts, generate one or morehypothesis based on the decomposed question and results of a primarysearch of answer sources, perform hypothesis and evidence scoring basedon a retrieval of evidence from evidence sources, perform synthesis ofthe one or more hypothesis, and based on trained models, perform a finalmerging and ranking to output an answer to the input question along witha confidence measure.

Some traditional QA system may engage Subject Matter Experts (SMEs) tocreate the ground truth (candidate QA pairs) as part of QA systemtraining. Generally, the process for this may have a unique set ofquestions that are given to each SME to create this set. In othersystems, a collaborative model may be used where each SME is given thesame questions and then through a group decision (e.g., voting) come upwith the most popular answers that are used to create the set. However,there may be example situations where, e.g., there are 2 or more equallygood answers that may vary depending on which end user is asking thequestion. This may be especially true in systems that may be deployed ortargeted to a wide range of end customers (e.g., end users in differentgeographical regions with added challenge of cultural sensitivities,and/or end users that may belong to various age groups, such as GenX vs.Baby Boomers vs. Millennials, and/or a group of users that may havedifferent work experience like veterans from the Vietnam War vs. theGulf War).

The Response Process:

As discussed above and referring also at least to FIGS. 3-6, responseprocess 10 may receive 300, by a computing device, a question from auser. Response process 10 may identify 302 a first answer provided by afirst subject matter expert. Response process 10 may identify 304 asecond answer provided by a second subject matter expert. Responseprocess 10 may determine 306 that a profile of the user matches a firstprofile of the first subject matter expert more than a second profile ofthe second subject matter expert. Response process 10 may send 308 thefirst answer provided by the first subject matter expert to the userwith a preference over the second answer provided by the second subjectmatter expert based upon, at least in part, determining 306 that theprofile of the user matches the first profile of the first subjectmatter expert more than the second profile of the second subject matterexpert.

As will be discussed below, in some implementations, response process 10may be applied, e.g., at least during the Ground Truth creation, whichmay enable teaching or training QA application 20/response process 10 toidentify cases where there may be equally valid answers, and provideinfluencing weight/score based on mapping Machine Models (e.g., whichmay reflect the features/attributes of the end user typically by using aSubject Matter Expert (SME) as the representative end user). Responseprocess 10 may be useful in many situations, including when providingthe “correct” answer also requires the most appropriate answer, e.g.,when two users are asking the same question but from different personal,experiential, and/or geographical situations.

For instance, assume for example purposes only that a user (e.g., user46) asks a question (e.g., via response process 10, QA application 20,client application 22, or combination thereof). In the example, assumethe question is “What are insurance benefits and features for warveterans?”. In the example, response process 10 may receive 300 thequestion from user 46.

In some implementations, response process 10 may create one or moreprofiles for one or more of the above-noted SMEs and/or user 46. Forexample, response process 10 may create certain models (e.g., targetedend user Personality/Experience/Trait (PET) models). The personalityportion of the PET model may indicate such things as, e.g.,psychological attributes, altruism, excitability, generosity, openness,imaginative, political, sympathetic, modest, cheerfulness, etc. Theexperience portion of the PET model may indicate, e.g., profession, workexperience, military service, participation in past events, resumes,geographic location, etc. The trait portion of the PET model mayindicate, e.g., social network behavior, shopping habits, culture,demographics, hobbies, etc. In some implementations, response process 10may gather this information about user 46 and/or the SMEs from varioussources (e.g., from registration data, social media profile information,blogs, etc., or any other publically available information). It will beappreciated that other models may be used without departing from thescope of the disclosure.

In some implementations, as will be discussed below, e.g., during theabove-noted training, each feature/attribute may be considered as aranking factor, where the SMEs (assuming they do represent one or moreof these features) may be creating training sets that are attached withthe above features/attributes as a key determinant. Hence, as will bediscussed below, the same question may be mapped to different answersbased on which profile attribute is given more weight. As will beappreciated, more complex models having numerous attributes withinterdependent correlations may be created with a range of weights thatmay affect the end answer. As will be discussed below, when such atrained system is deployed by response process 10, an end user (e.g.,user 46) may be classified based on available information, and once theset of answers are generated, response process 10 may do a mapping ofuser 46 to one of the trained PET models and based on that, rank therelated answers higher leading to a “Most Relevant Answer” for user 46.

As will be discussed below, in some implementations, if at least one ofthe SMEs is a real representative of the targeted end user (e.g., user46 who is asking the question), response process 10 may automaticallycreate the models from that particular SME's profile. If at least one ofthe SMEs are not real representatives of user 46, response process 10may create the appropriate PET model manually based on, e.g., theapplication domain, by generating features/weights that may be part ofthat PET model which may affect the final ranking or prioritizing of theanswers.

For instance, and continuing with the above example, assume for examplepurposes only that response process 10 has created a profile (e.g., PETprofile) for user 46, which indicates an example attribute that user 46is a veteran of the Iraq war. Further assume that response process 10has created a PET profile for a first SME, which indicates an exampleattribute that the first SME was also a veteran of the Iraq war. Furtherassume that response process 10 has created a PET profile for a secondSME, which indicates an example attribute that the second SME was aveteran of the Gulf war. The nature of the priorities in terms ofinsurance benefits and features may be different in each case. Thus,when response process 10 creates each PET model, differentcharacteristics may be considered. For instance, the described examplemay consider a slightly complex characteristic as the consideration ofwhich is the nature of injuries or health issues based on each war.

In some implementations, response process 10 may identify 302 a firstanswer provided by the first SME. For example, recall that the first SMEhas a PET model that indicates an example attribute that the first SMEwas also a veteran of the Iraq war. In the example, the answer to “whatare insurance benefits and features for war veterans?” provided by thefirst SME may include particularly relevant information about insurancebenefits and features for Iraq war veterans (e.g., information aboutinsurance benefits and features for post traumatic stress disordercombined with amputations).

In some implementations, response process 10 may identify 304 a secondanswer provided by a second subject matter expert. For example, recallthat the second SME has a PET model that indicates an example attributethat the second SME was a veteran of the Gulf war. In the example, theanswer to “what are insurance benefits and features for war veterans?”provided by the second SME may include particularly relevant informationabout insurance benefits and features for Gulf war veterans (e.g.,information about insurance benefits and features for an unexplainedillness known as “Gulf War Syndrome” and Amyotrophic Lateral Sclerosis).

In some implementations, response process 10 may determine 306 that aprofile of the user matches a first profile of the first subject matterexpert more than a second profile of the second subject matter expert.For instance, and continuing with the above example, further assume thatresponse process 10 has generated a profile for user 46 that indicatesan example attribute that user 46 was a veteran of the Iraq war.

In some implementations, determining 306 that the profile of the usermatches the first profile of the first subject matter expert more thanthe second profile of the second subject matter expert may includeresponse process 10 comparing 310 the profile of the user with the firstprofile of the first subject matter expert, and may include comparing312 the profile of the user with the second profile of the secondsubject matter expert. For example, response process 10 may compare 310the profile of user 46 with the profile of the first SME, and maycompare 312 the profile of user 46 with the profile of the second SME.As noted above, both the profile of user 46 and the profile of the firstSME share a common feature/attribute of having served in the Iraq war.As such, in the example, response process 10 may determine 306 that theprofile of user 46 matches the profile of the first SME more than theprofile of the second SME.

It will be appreciated that while the disclosure is described using 2SME's with different professional experiences, other examplefeature/attributes may be used as well without departing from the scopeof the disclosure. For instance, in some implementations, the SMEs maybe from different cultural backgrounds, and therefore may each expressequally “correct” answers, but with a culturally-attuned sensitivity topoliteness, social protocol, regional dialect, etc. For example, a thirdand fourth SME may both work in the same role for the same company(e.g., telecomm solution architect, insurance underwriter, etc.) but onemay live in Japan and one may live in the US. In the example, assumefurther that a profile of a third SME shows the third SME lives in Japanand the fourth SME lives in the US (and both work for the same telecommsolution architect company). In the example, assuming for examplepurposes only that a profile of user 48 indicates that user 48 lives inJapan, response process 10 may compare 310 the profile of user 48 withthe profile of the third SME, and may compare 312 the profile of user 48with the profile of the fourth SME. As noted above, both the profile ofuser 48 and the profile of the third SME share a commonfeature/attribute of living in Japan. As such, in the example, responseprocess 10 may determine 306 that the profile of user 48 matches theprofile of the third SME more than the profile of the fourth SME. Assuch, the use of the particular feature/attributes being professionalexperiences should be taken as an example only and not to limit thescope of the disclosure.

It will be appreciated that the use of only focusing on a singleattribute in a profile is for simplicity purposes only and should not betaken to limit the scope of the disclosure. For example, each profilemay include multiple attributes that may be used to determine 306 whichSME's profile better matches the profile of user 46. For example,determining 306 that the profile of the user matches the first profileof the first subject matter expert more than the second profile of thesecond subject matter expert may include response process 10 applying314 a weight to one or more features of the profile. For instance, asnoted above, at least one of the user profile, the first profile, andthe second profile may include a personality model. At least one of theuser profile, the first profile, and the second profile may include anexperience model. At least one of the user profile, the first profile,and the second profile may include a trait model. As also noted above,the personality portion of the PET model may indicate such things as,e.g., altruism, excitability, generosity, openness, imaginative,political, sympathetic, modest, cheerfulness, etc. The experienceportion of the PET model may indicate, e.g., profession, workexperience, military service, resumes, geographic location, etc. Thetrait portion of the PET model may indicate, e.g., social networkbehavior, shopping habits, culture, etc.

As such, assume for example purposes only that the profile of user 46includes altruism, Iraq war veteran, and activity on gardening socialmedia pages. Further assume that the profile for the first SME includesgenerosity, Iraq war veteran, and activity on model plane design socialmedia pages, while the profile for the second SME includes cheerfulness,Gulf war veteran, and activity on gardening social media pages. In theexample, further assume that response process 10 has placed a higherweight on the feature trait portion of the profile for user 46 (e.g.,Iraq war veteran), and a lower weight on the remaining feature portions.As such, in the example, because the higher weight is placed on thetrait portion (which is matched to the second SME's profile), ratherthan the experience portion (which is matched to the first SME'sprofile), response process 10 may determine 306 that the profile of user46 matches the profile of the second SME more than the profile of thefirst SME. As such, any combination of profile features may be used(with or without weights) without departing from the scope of thedisclosure. In some implementations, the weight may be applied 314 toeach feature/attribute based upon correlation. For example, as theabove-noted question includes the terms “war” or “veterans”, responseprocess 10 may automatically apply 314 a higher weight to thefeature/attribute “military service” experience portion, since there maybe a correlation between the terms “war” or “veterans” and “Iraqveteran”. It will be appreciated that other techniques to determine theappropriate weight may be used without departing from the scope of thedisclosure.

In some implementations, response process 10 may send 308 the firstanswer provided by the first subject matter expert to the user with apreference over the second answer provided by the second subject matterexpert based upon, at least in part, determining 306 that the profile ofthe user matches the first profile of the first subject matter expertmore than the second profile of the second subject matter expert. Forinstance, using the above-noted ground truth training along with themapped PET Models, when user 46 asks the above-noted question, responseprocess 10 may generate a real-time PET Model for user 46 from theirprofile. In some implementations, this may require a combination ofNLP/Machine Models. Once response process 10 comes up with the candidateset of answers that have a mapping to pre-trained PET models, during theanswer-scoring/merging, the “real” Pet Model from user 46 may be takeninto consideration to provide a higher score for answers that may have anear match with the mapped PET MODELS. This may enable the outputanswers that match or are more relevant to user 46 to be sent 308. Assuch, response process 10 may utilize the concurrence/similarity ofattributes between the profile of user 46 and the profile of the SME'swith similar or parallel attributes as an additional information streamwhen scoring/ranking candidate answers.

For instance, continuing with the above example, recall that the answerto “what are insurance benefits and features for war veterans?” providedby the first SME may include particularly relevant information aboutinsurance benefits and features for Iraq war veterans (e.g., informationabout insurance benefits and features for post traumatic stress disordercombined with amputations), as the first SME's profile includes theattribute of having served in the Iraq war, and further recall that theanswer to “what are insurance benefits and features for war veterans?”provided by the second SME may include particularly relevant informationabout insurance benefits and features for Gulf war veterans (e.g.,information about insurance benefits and features for an unexplainedillness known as “Gulf War Syndrome” and Amyotrophic Lateral Sclerosis),as the second SME's profile includes the attribute of having served inthe Gulf war. Thus, in the example, because response process 10 hasdetermined 306 that the profile of user 46 matches the profile of thefirst SME more than the second SME, response process 10 may send 308 touser 46 the answer provided by the first subject matter expert (e.g.,particularly relevant information about insurance benefits and featuresfor Iraq war veterans for post traumatic stress disorder combined withamputations) with a preference over sending to user 46 the second answerprovided by the second subject matter expert (e.g., particularlyrelevant information about insurance benefits and features for Gulf warveterans for an unexplained illness known as “Gulf War Syndrome” andAmyotrophic Lateral Sclerosis). As such, in the example, the first SME'sanswer may be equally as accurate as the second SME's answer, however,the first SME's answer may be more appropriate for user 46 than thesecond SME's answer. In the example, during the Ground Truth creation,if the SMEs are coming up with answers that are different, but equallyrelevant based on who is asking the question, response process 10 maymap the SME's to a particular PET Model (e.g., mapping the first SME toan Iraq war based PET Model and mapping the second SME to a Gulf warbased PET Model).

Conversely, assume for example purposes only that another user (e.g.,user 48) asks response process 10 the same question, where responseprocess 10 has generated a profile for user 48 that indicates an exampleattribute that user 48 was a veteran of the Gulf war. In the example,even though the same question was asked, response process 10 may send308 to user 48 the answer provided by the second subject matter expert(e.g., particularly relevant information about insurance benefits andfeatures for Gulf war veterans for an unexplained illness known as “GulfWar Syndrome” and Amyotrophic Lateral Sclerosis) with a preference oversending to user 46 the answer provided by the first subject matterexpert (e.g., particularly relevant information about insurance benefitsand features for Iraq war veterans for post traumatic stress disordercombined with amputations). As such, in the example, the second SME'sanswer may be equally as accurate as the first SME's answer, however,the second SME's answer may be more appropriate for user 48 than thefirst SME's answer.

In some implementations, the preference may include a ranking of thefirst answer provided by the first subject matter expert higher than thesecond answer provided by the second subject matter expert. Forinstance, referring at least to FIG. 4, and continuing with the examplewhere the above-noted question is asked by user 46, response process 10may send 308 to user 46 a list of answers (e.g., with supportingreference documents) for both the first SME's answers and the secondSME's answers. In the example, the answers provided by the first SME maybe ranked higher than the answers provided by the second SME. The listis shown via an example user interface 400 associated with responseprocess 10 on display 212. It will be appreciated that response process10 may send 308 the answers in any format using any known technique,such as email, text, web page, etc.

It will be appreciated that ranking of the first answer provided by thefirst subject matter expert higher than the second answer provided bythe second subject matter expert may include response process 10 onlysending 308 the answers provided by the first SME without sending theanswers provided by the second SME. For instance, and referring at leastto FIG. 5, response process 10 may send 308 to user 46 a list of answers(e.g., with supporting reference documents) for the first SME's answerswithout sending 308 the second SME's answers. In some implementations,general answers provided by other SMEs may also be displayed. In theexample, the answers provided by the first SME may be ranked higher thanthe answers provided by the “general” SME. For example, in a generalizedsystem (e.g., without a mapped SME), the question from user 46 maysimply result in a list of general answers with supporting referencedocuments for general inpatient and outpatient services. As such, in theexample, general answers may still be sent to user 46 along with (butwith a lower rank than) the answers provided by the first SME. The listis shown via an example user interface 500 associated with responseprocess 10 on display 212. It will be appreciated that any method ofsending 308, displaying and/or ranking answers may be used withoutdeparting from the scope of the disclosure.

In some implementations, and referring at least to FIG. 6, responseprocess 10 may determine the top answers to user 46's question (as maybe done in conventional QA systems without mapped SMEs). Afterdetermining the top answers, response process 10 may then rank orre-rank the top answers according to the above-noted PET model andselect procedures that may be most appropriate, which may then be sent308 to user 46.

It will be appreciated that while the disclosure is described using a QAsystem, response process 10 may be adapted to work with any exampleanswer query/response systems (e.g., conventional data store or Internetsearch engines). As such, the use of a QA system should be taken asexample only and not to otherwise limit the scope of the disclosure.

The terminology used herein is for the purpose of describing particularimplementations only and is not intended to be limiting of thedisclosure. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps (notnecessarily in a particular order), operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps (not necessarily in a particular order),operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements that may be in the claims below areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present disclosure has been presentedfor purposes of illustration and description, but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications, variations, and any combinations thereof will be apparentto those of ordinary skill in the art without departing from the scopeand spirit of the disclosure. The implementation(s) were chosen anddescribed in order to best explain the principles of the disclosure andthe practical application, and to enable others of ordinary skill in theart to understand the disclosure for various implementation(s) withvarious modifications and/or any combinations of implementation(s) asare suited to the particular use contemplated.

Having thus described the disclosure of the present application indetail and by reference to implementation(s) thereof, it will beapparent that modifications, variations, and any combinations ofimplementation(s) (including any modifications, variations, andcombinations thereof) are possible without departing from the scope ofthe disclosure defined in the appended claims.

What is claimed is:
 1. A computer-implemented method comprising:receiving, by a computing device, a question from a user; generating,after the question is received, a real-time Personality/Experience/Trait(PET) Model from a profile of the user; identifying a first answerprovided by a first subject matter expert; identifying a second answerprovided by a second subject matter expert; mapping, after the firstanswer and the second answer are identified, the user to at least onetrained PET Model created by the at least one of the first subjectmatter expert and the second subject matter expert; determining that aprofile of the user matches a first profile of the first subject matterexpert more than a second profile of the second subject matter expert,wherein determining that the profile of the user matches the firstprofile of the first subject matter expert more than the second profileof the second subject matter expert includes: applying a weight to oneor more features of the profile of the user, the first profile of thefirst subject matter expert, and the second profile of the secondsubject matter expert, wherein applying the weight to the one or morefeatures is based upon, at least in part, determining a correlationbetween the one or more features of the profile of the user and thefirst profile of the first subject matter expert, wherein applying theweigh to the one or more features of the profile is determined basedupon a similarity of attributes between the real-time PET Model relativeto the user and the trained PET Model relative to the at least onsubject matter expert; and upon determining the correlation between theone or more features of the profile of the user and the first profile ofthe first subject matter expert, applying a higher weight to the one ormore features having the correlation between the profile of the user andthe profile of the first subject matter expert; and sending to the userthe first answer provided by the first subject matter expert with apreference over the second answer provided by the second subject matterexpert based upon, at least in part, determining that the profile of theuser matches the first profile of the first subject matter expert morethan the second profile of the second subject matter expert, wherein thefirst answer provided by the first subject matter expert is moreappropriate than the second answer provided by the second subject matterexpert based upon the correlation between the one or more features ofthe profile of the user and the first profile of the first subjectmatter expert.
 2. The computer-implemented method of claim 1 whereindetermining that the profile of the user matches the first profile ofthe first subject matter expert more than the second profile of thesecond subject matter expert includes: comparing the profile of the userwith the first profile of the first subject matter expert; and comparingthe profile of the user with the second profile of the second subjectmatter expert.
 3. The computer-implemented method of claim 2 wherein atleast one of the user profile, the first profile, and the second profileincludes a personality model.
 4. The computer-implemented method ofclaim 2 wherein at least one of the user profile, the first profile, andthe second profile includes an experience model.
 5. Thecomputer-implemented method of claim 2 wherein at least one of the userprofile, the first profile, and the second profile includes a traitmodel.
 6. The computer-implemented method of claim 1 wherein thepreference includes a ranking of the first answer provided by the firstsubject matter expert higher than the second answer provided by thesecond subject matter expert.
 7. A computer program product residing ona non-transitory computer readable storage medium having a plurality ofinstructions stored thereon which, when executed by a processor, causethe processor to perform operations comprising: receiving a questionfrom a user; generating, after the question is received, a real-timePersonality/Experience/Trait (PET) Model from a profile of the user;identifying a first answer provided by a first subject matter expert;identifying a second answer provided by a second subject matter expert;mapping, after the first answer and the second answer are identified,the user to at least one trained PET Model created by the at least oneof the first subject matter expert and the second subject matter expert;determining that a profile of the user matches a first profile of thefirst subject matter expert more than a second profile of the secondsubject matter expert, wherein determining that the profile of the usermatches the first profile of the first subject matter expert more thanthe second profile of the second subject matter expert includes:applying a weight to one or more features of the profile of the user,the first profile of the first subject matter expert, and the secondprofile of the second subject matter expert, wherein applying the weightto the one or more features is based upon, at least in part, determininga correlation between the one or more features of the profile of theuser and the first profile of the first subject matter expert, whereinapplying the weigh to the one or more features of the profile isdetermined based upon a similarity of attributes between the real-timePET Model relative to the user and the trained PET Model relative to theat least on subject matter expert; and upon determining the correlationbetween the one or more features of the profile of the user and thefirst profile of the first subject matter expert, applying a higherweight to the one or more features having the correlation between theprofile of the user and the profile of the first subject matter expert;and sending to the user the first answer provided by the first subjectmatter expert with a preference over the second answer provided by thesecond subject matter expert based upon, at least in part, determiningthat the profile of the user matches the first profile of the firstsubject matter expert more than the second profile of the second subjectmatter expert, wherein the first answer provided by the first subjectmatter expert is more appropriate than the second answer provided by thesecond subject matter expert based upon the correlation between the oneor more features of the profile of the user and the first profile of thefirst subject matter expert.
 8. The computer program product of claim 7wherein determining that the profile of the user matches the firstprofile of the first subject matter expert more than the second profileof the second subject matter expert includes: comparing the profile ofthe user with the first profile of the first subject matter expert; andcomparing the profile of the user with the second profile of the secondsubject matter expert.
 9. The computer program product of claim 8wherein at least one of the user profile, the first profile, and thesecond profile includes a personality model.
 10. The computer programproduct of claim 8 wherein at least one of the user profile, the firstprofile, and the second profile includes an experience model.
 11. Thecomputer program product of claim 8 wherein at least one of the userprofile, the first profile, and the second profile includes a traitmodel.
 12. The computer program product of claim 7 wherein thepreference includes a ranking of the first answer provided by the firstsubject matter expert higher than the second answer provided by thesecond subject matter expert.
 13. A computing system including aprocessor and a memory configured to perform operations comprising:receiving a question from a user; generating, after the question isreceived, a real-time Personality/Experience/Trait (PET) Model from aprofile of the user; identifying a first answer provided by a firstsubject matter expert; identifying a second answer provided by a secondsubject matter expert; mapping, after the first answer and the secondanswer are identified, the user to at least one trained PET Modelcreated by the at least one of the first subject matter expert and thesecond subject matter expert; determining that a profile of the usermatches a first profile of the first subject matter expert more than asecond profile of the second subject matter expert, wherein determiningthat the profile of the user matches the first profile of the firstsubject matter expert more than the second profile of the second subjectmatter expert includes: applying a weight to one or more features of theprofile of the user, the first profile of the first subject matterexpert, and the second profile of the second subject matter expert,wherein applying the weight to the one or more features is based upon,at least in part, determining a correlation between the one or morefeatures of the profile of the user and the first profile of the firstsubject matter expert, wherein applying the weigh to the one or morefeatures of the profile is determined based upon a similarity ofattributes between the real-time PET Model relative to the user and thetrained PET Model relative to the at least on subject matter expert; andupon determining the correlation between the one or more features of theprofile of the user and the first profile of the first subject matterexpert, applying a higher weight to the one or more features having thecorrelation between the profile of the user and the profile of the firstsubject matter expert; and sending to the user the first answer providedby the first subject matter expert with a preference over the secondanswer provided by the second subject matter expert based upon, at leastin part, determining that the profile of the user matches the firstprofile of the first subject matter expert more than the second profileof the second subject matter expert, wherein the first answer providedby the first subject matter expert is more appropriate than the secondanswer provided by the second subject matter expert based upon thecorrelation between the one or more features of the profile of the userand the first profile of the first subject matter expert.
 14. Thecomputing system of claim 13 wherein determining that the profile of theuser matches the first profile of the first subject matter expert morethan the second profile of the second subject matter expert includes:comparing the profile of the user with the first profile of the firstsubject matter expert; and comparing the profile of the user with thesecond profile of the second subject matter expert.
 15. The computingsystem of claim 14 wherein at least one of the user profile, the firstprofile, and the second profile includes a personality model.
 16. Thecomputing system of claim 14 wherein at least one of the user profile,the first profile, and the second profile includes an experience model.17. The computing system of claim 14 wherein at least one of the userprofile, the first profile, and the second profile includes a traitmodel.